SEPT: Towards Scalable and Efficient Visual Pre-training

Recently, the self-supervised pre-training paradigm has shown great potential in leveraging large-scale unlabeled data to improve downstream task performance. However, increasing scale of real-world scenarios requires prohibitive computational costs and faces challenge uncurated samples. To address these issues, we build a task-specific framework from selection perspective based on simple hypothesis that samples with similar distribution target can bring substantial performance gains. Buttressed by hypothesis, propose first yet novel for Scalable Efficient visual Pre-Training (SEPT) introducing retrieval pipeline selection. SEPT leverage pre-trained model extract features entire dataset initialization. Then, specific task, retrievals most feature similarity each instance pre-training. Finally, pre-trains selected manner finetuning. By decoupling available upstream achieves high scalability efficiency pre-training, resulting architecture flexibility. Results various tasks demonstrate achieve competitive or even better compared ImageNet while reducing size training one magnitude without resorting any extra annotations.